Field of the Invention
The present invention relates in general to the field of electronics, and more specifically to computer software testing and selectively executing test cases to reduce overall test case execution.
Description of the Related Art
Testers often test computer software using a suite of test cases. Test cases are also often referred to as test scripts. The test cases often include a set of conditions or variables, and the tester tests the computer software in accordance with the test cases. Generally, the test case provides a known input with an expected output, and the tester determines if the computer software passes or fails based on whether the computer software respectively produced the expected output or another output.
Testers test computer software when certain conditions occur. An exemplary condition is when computer software is modified either through the addition of new code or the revision of existing code. Regression testing tests the computer software in accordance with test cases to determine whether the modified computer software produces the correct predetermined behavior despite the changes made to the code. Since computer software frequently change during the process of development for initial and new versions, regression testing also often occurs frequently. Any product that invests in exhaustive regression testing invariably has a very large automated regression test suite of test cases. The objective of this test suite is that it can be executed with every code modification to identify potential regressions, i.e. errors.
FIG. 1 depicts computer software development and testing system 100, which performs exhaustive test case regression testing. A software developer utilizes computer 102 to develop code 104. From computer 102, the developer performs a code check-in 106 to check the code 104 into computer system 108. The code 104 is all or part of software 110. Computer system 108 includes a source control system 109 that assigns a new revision number to the software 110 to provide version control and stores each version of the software 110 in a file versioning repository memory 111 of the computer system 108. The source control system 109 sends CHECK_IN data to notify a regression testing system 111 of computer system 113 when code 104 is checked-in. The CHECK_IN data provides specific details associated with the code check-in. Exemplary CHECK_IN data includes:                Identification of the user who checked in the code 104;        The files of code 104 that were checked in;        The check in time;        The assigned revision number; and        Identification of the repository in which the code 104 was checked in.        
The source control system 109 is a software program executed by the computer system 108, and the regression testing system 111 is a software program executed by the computer system 113.
The regression testing system 111 performs regression testing to determine whether the code 104 caused any functional or other errors in the software 110. Regression testing can involve one or more large test suites that typically have thousands of test cases and can take between 2 and 30 minutes each to run. Regression testing system and process 112 includes a test suite 114, and test suite 114 includes N test cases, where N is an integer representing the number test cases in test suite 114. The regression system and process 112 is incorporated as data and applications in computer system 108. Regression testing process 116 runs each of the N test cases in test suite 114. For large values of N, such as greater than 100, when each test case 1 through N runs serially in computer system 108, the regression test suite 114 may take in excess of 48 hours to run. Running regression test suite 114 serially can, thus, take a significant amount of time. Utilizing parallel processing, by adding one or more additional computer systems to computer system 108 to run all or a subset of the test cases in parallel, reduces regression testing time but can increase costs significantly. The results data 118 contains the results of the regression testing.
Thus, conventional regression testing is costly either in terms of time, cost, or both. However, without regression testing, computer software can produce errors that may not be caught and/or understood until actual deployment.